Assessment of Prediction Techniques: The Impact of Human Uncertainty

Many data mining approaches aim at modelling and predicting human behaviour. An important quantity of interest is the quality of model-based predictions, e.g. for €nding a competition winner with best prediction performance. In real life, human beings meet their decisions with considerable uncertainty. Its assessment and resulting implications for statistically evident evaluation of predictive models are in the main focus of this contribution. We identify relevant sources of uncertainty as well as the limited ability of its accurate measurement, propose an uncertainty-aware methodology for more evident evaluations of data mining approaches, and discuss its implications for existing quality assessment strategies. Speci€cally, our approach switches from common point-paradigm to more appropriate distributionparadigm. Œis is exempli€ed in the context of recommender systems and their established metrics of prediction quality. Œe discussion is substantiated by comprehensive experiments with real users, largescale simulations, and discussion of prior evaluation campaigns (i.a. Netƒix Prize) in the light of human uncertainty aspects. Kevin Jasberg and Sergej Sizov. . Assessment of Prediction Techniques: Œe Impact of Human Uncertainty. In Proceedings of , , , 10 pages. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for pro€t or commercial advantage and that copies bear this notice and the full citation on the €rst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permiŠed. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speci€c permission and/or a fee. Request permissions from [email protected].